Scoring the Forecaster by Mean Resulting Payoff of a Distribution of Decision Problems

We present a new way to understand and characterize the choice of scoring, objective , or loss function for a model, machine, or expert estimating (i.e. predicting) the probabilities of events or outcomes (such as class membership). The ultimate value of a probability estimate lies in the actual payoo (utility) accruing to those who use this information to make a decision. We allow that we often cannot specify with certainty that the estimate will be used in a particular decision problem, characterized by a particular decision-outcome payoo matrix (cost schedule), and thus by a particular decision threshold. Instead, we consider the more general case of a distribution over such matrices. The proposed scoring function is the expectation, with respect to this distribution, of the payoo that will actually be received. Square-error loss (or Brier score) and log-likelihood (or various entropic measures) arise from speciic examples of such distributions, and even common single-threshold measures such as the misclassiication rate obtain from degenerate special cases. Our scoring functions are always \honesty-rewarding," or maximized in expectation when the estimate equals the true outcome probability. There is still no single \universal" choice of scoring function appropriate in all cases, but speciic choices correspond to speciic assumptions about the probability that the predictions will be thresholded in various ways when used to make a decision.